1. Field
The invention relates to a method for determining static elements in image sources, a corresponding system, imaging device, a movable device comprising such an imaging device and a corresponding computer program product. In particular the analysis of images relates to detecting static flaws in images out of a sequence of images captured by an imaging means. The method is preferably used in vehicles that are equipped with driver assistance systems taking advantage of image processing, or autonomous vehicles.
2. Description of Related Art
The invention is in the field of autonomous devices relying on computer generated vision or other sensing techniques perceiving the surrounding of a vehicle on which such a system is mounted. An example for an autonomous device for applying the invention is an autonomous lawn mower but the invention is by no means limited thereto. In order to act autonomously with high reliability it is important that these autonomous devices perceive their surrounding with a certain detail under all circumstances. Imaging means such as cameras are increasingly employed in this area and provide capabilities to the autonomous devices related to human abilities such as distance estimation, street lane detection or complex object and people detection. Nevertheless it is of utmost importance to the overall system reliability that optical sensors such as cameras provide their image signals even under adverse conditions encountered in outdoor applications. Optical sensors and in particular their lenses become dirty or a lens of an optical sensor might receive scratches. Accordingly regular maintenance or cleaning the sensors becomes increasingly important. Hence it is also important to provide for autonomous movable devices to perform self-checks on optical sensors in order to generate information for suitable measures to counter the problem of reduced vision of the sensors and thus to avoid false analysis of the captured image.
The term “autonomous movable device” is commonly known in the art as referring to an unmanned device which has a drive means or propulsion means in order to move the autonomous device (“self-propelled device”), an onboard energy reservoir to power the propulsion means, one or more sensors and a control means functionally connected to the sensor(s) and the drive means. The autonomous device navigates in a free manner without human support based on sensor data acquired by the sensor(s) and processed in the control means in order to generate control signals for the propulsion means.
In the future for improving the functionality of autonomously operating movable devices particularly in outdoor environment more sensors and/or sensors with improved characteristics or altogether new capabilities such as optical sensors to the autonomous device have to be expected. However the quality of signals generated by these new sensors and in particular by optical sensors, e.g. cameras, depends on clean, lenses and transparent covers in the line of vision for acquiring visual information in a demanding outdoor environment. An outdoor environment, such as a garden under varying weather conditions, and performing tasks, such as cutting grass in case of an autonomous lawn mower, further increases the exposure of the sensor to dirt, sand, parts of plants, water, etc. Hence regular maintenance of the sensors is of high importance to avoid failure of the autonomous device due to sensor failure or faulty analysis.
The invention is particularly advantageous, when the sensor means comprises one or more optical sensors such as cameras. The use of cameras in the outdoor environment provides the autonomous movable device with an enhanced control capability, but the vision of cameras on the other hand significantly decreases with the dirt on a lens of a camera. Hence navigation and other operational characteristics of the autonomous robot device exposed to dirt profits from timely sensor cleaning.
The autonomous movable device of a preferred embodiment of the present invention is an autonomous lawn mower. The autonomous lawn mower also comprises working means including one or more blades for cutting grass and tends to soil rapidly with the cut grass adhering to any surface of the autonomous lawn mower when operating.
European Patent document EP1 983 334 B1 addresses the problem of detecting dust on a frontal lens of an optical system such as a surveillance camera. Additional optics generate sub-images in such a manner that the light rays generating these sub-images pass different positions in the front lens. The proposal in EP1 983 334 B1 needs additional technical means for detecting dust and flaw and is therefore complex and expensive to implement.
Patent document U.S. Pat. No. 6,861,636 B2 discloses a stereo camera setup to detect moisture on the surface of sensor arrays, e.g. for controlling wipers for a windshield of a vehicle. However two image sensors are required in order to generate stereoscopic depth and therefore introduce additional hardware and hence complexity into the sensor system for detecting moisture on the windshield.
US Patent document U.S. Pat. No. 7,636,114 B2 shows how to detect a pixel position corresponding to dust on the surface on an imaging sensor and its optical system. A special test setup is used to capture an image with uniform luminance. Then the intensity of each pixel is compared to a threshold which defines an expected intensity for the given uniform luminance. As uniform luminance is hardly found in a natural environment, the particular test setup is essential to the disclosed method and the method therefore ill suited to supervising a lens of a sensor during normal operation of a vehicle or autonomous movable device.
Patent application publication DE 10 2007 057 745 A1 discloses a process for controlling the wipers for a windshield of a vehicle by detecting objects on the windscreen. Objects on the windscreen are distinguished from objects in the background by capturing two images at different times and comparing the two images with each other. In the described embodiment the acquired images are transformed into edge images and for the purpose of comparison are multiplied pixel-wise in order to enhance objects such as dirt on the windscreen present in both images. If there are objects detected on the windscreen, a spray unit is activated for cleaning the windscreen. In the embodiment the single first edge image is analyzed for raindrops based on the specific structure of raindrops and the control of the wiper assembly is started based on the analysis result of the first image.
Parameters for generating edge images heavily depend on light intensity of the pixels. These parameters are complex in a diverse and rapidly changing light condition such as encountered by vehicles moving in real world environment and especially during night, or for autonomous lawn mowers operating in gardens or autonomous cleaning robots operating indoors. Further a vehicle employing the system according DE 10 2007 057 745 A1 might stand at red traffic lights, or cruise on a straight road with wide view over far distance. Given these circumstances, static objects on the windshield will be difficult to distinguish from the background which is also static under these conditions and the static object detection provide only results of limited reliability.
Taking the state of the art into consideration, the technical problem of determining the presence of static elements on the lens of an image or video source needs to be addressed overcoming the abovementioned disadvantages of the state of the art.